Monitoring systems for online environments generally track characteristics of a particular operating system or process, and generate data describing a characteristic of the operating system or process. For example, a sensor may sense a characteristic of an operating process and generate a reading every second, or many times a second. Such data will then generally be stored for subsequent analysis. This generally results in large amounts of data to be stored and analyzed, which can be time-consuming and inefficient. Accordingly, it is desirable to compress such data for storage and to reduce storage capacity. In addition, very small measurement changes (e.g., measurement noise or repeated values) may not be meaningful and are not needed in long-term storage.
Such types of monitoring systems are particularly applicable in large industrial systems for which monitoring of changes in an industrial process can be important. In such cases, compression of data may be required due to the sheer amount of data collected, while preservation of features in the compressed data is also generally important, because it allows analysis to understand and improve the industrial process.
There are currently several methods to compress process data in an online environment. The two most common methods are the boxcar method and the swinging door method. Both methods have drawbacks. For example, both methods require substantial user input at set-up time to select specific data compression parameters to be applied during a compression operation. Even with such user input, these methods have relatively low levels of compression (e.g., as low as 20-30%). In some circumstances, these existing methods may apply a common set of compression parameters across an entire data stream, comprised of different types of measurements, which may result in over- or under-compression.
Other existing systems will operate based on pre-stored data to perform data compression in a “batch” mode. Such systems, by their very nature, require substantial storage resources to allow for storage of pre-compressed data, and also introduce delay between a time when data is received from a sensor or tag, and a time when compressed data is available for analysis. Other solutions, although real-time in compression, may require later reconstruction of data for analysis, introducing an extra step.
Therefore, a need exists in the art for an improved manner of performing data compression.